External Encapsulation

  An alternative communications backend for R, implementing a request by R-Core at R Project Sprint 2023

  Next generation completely event-driven promises

  Asynchronous parallel / distributed backend

  Asynchronous parallel / distributed backend

  Host ADBC database connections in daemon processes

  Seamless cross-process use of Torch tensors and models

Re-encapsulating mirai

Why crew?

• Extends mirai to distributed computing environments.
• Centralized R6 interface for tasks.
• Worker auto-scaling to respond fluctuating task loads.

Moju Kapu in crew

.pull-left[

Encapsulation

• Uses mirai developer interface: daemon(), nextget(), saisei() etc.
• R6 class system for the controller interface.

]

.pull-right[

Modularity

• Plugin system to launch parallel workers on different environments.
• Docs guide users to write their own plugins.
• Existing encapsulated plugins for SLURM, AWS Batch, etc.

]

R6 classes

R6 object composition

mirai tasks, different interface

Different plugin, different controller

Write your own launcher plugin

• Tutorial to write plugins: https://wlandau.github.io/crew/articles/plugins.html

Controller wrapper

• Tutorial to write plugins: https://wlandau.github.io/crew/articles/plugins.html

100x Faster

Setup:

library(mirai)
library(future)

d <- daemons(1, dispatcher = FALSE)
plan("multisession", workers = 1)

m <- mirai(1)
collect_mirai(m)
#> [1] 1

f <- future(1)
value(f)
#> [1] 1

^{Created on 2024-05-27 with reprex v2.1.0}

100x Faster (Cont’d)

Benchmarking:

bench::mark(mirai(1), future(1), relative = TRUE, check = FALSE)
#> # A tibble: 2 × 6
#>   expression   min median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <dbl>  <dbl>     <dbl>     <dbl>    <dbl>
#> 1 mirai(1)      1      1       74.6      1        1   
#> 2 future(1)   158.   113.       1        5.72     2.75

bench::mark(collect_mirai(m), value(f), relative = TRUE)
#> # A tibble: 2 × 6
#>   expression         min median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>       <dbl>  <dbl>     <dbl>     <dbl>    <dbl>
#> 1 collect_mirai(m)   1      1        84.1       Inf      NaN
#> 2 value(f)          79.3   89.0       1         NaN      Inf

^{Created on 2024-05-27 with reprex v2.1.0}

WYSIWYG Concept

Production usage requires ‘correctness’ over ‘convenience’.

Code behaves as written. There is no reliance on non-transparent static code analysis, which can result in inefficient behaviour, or even fail due to hidden global options:

library(mirai)
library(future)
df <- list(a = double(1e8), b = 1)

m <- mirai(2 * x, x = df$b)
m[]
#> [1] 2

f <- future(2 * df$b)
#> Error in getGlobalsAndPackages(expr, envir = envir, tweak =
#> tweakExpression, : The total size of the 1 globals exported for
#> future expression ('2 * df$b') is 762.94 MiB.. This exceeds the
#> maximum allowed size of 500.00 MiB (option 'future.globals.maxSize')
#> . There is one global: 'df' (762.94 MiB of class 'list')

One Million Promises

library(mirai)
daemons(8, dispatcher = FALSE)
#> [1] 8

r <- 0
start <- Sys.time()
m <- mirai_map(1:1000000, \(x) x, .promise = \(x) r <<- r + x)
Sys.time() - start
#> Time difference of 6.42396 mins

later::run_now()
r
#> [1] 500000500000

^{Created on 2024-05-27 with reprex v2.1.0
Running on an Intel i7 Gen 11 notebook with 16GB RAM.}